Complex Systems, Information Theory . . .

W odzis aw Duch and Norbert Jankowski In this paper relation between complex systems information theory and the simplest models of neural networks are elucidated Two di erent types of complex systems are distinguished new complexity measure based on the graph theory de ned hierarchy of the correlation matrices introduced and connection with the correlation matrix memories and other types of neural models explained Two Types of Complex Systems Complex system theory is a new eld of science that emerged at the end of the last decade A working de nition of complex systems was given at the conference devoted to this subject systems that exhibit complicated behavior but for which there is some hope that the underlying structure is simple in the sense of being governed by a small number of degrees of freedom Another working de nition is A system is loosely de ned as complex if it is composed of a large number of elements interacting with each other and the emergent global dynamics is qualitatively di erent from the dynamics of each one of the parts Examples of complex systems include fractals snow akes cellular automata games Ising and spin glass models arti cial neural nets and many dynamical sys tems that exhibit complex behavior starting from simple dynamics Other complex systems and objects such as language structure of words and sentences proteins genes visual data market analysis data do not t to these de nitions We do not even know if a simple dynamics responsible for their complexity exist and in case of many body systems such as proteins we are convinced that a large number of degrees of freedom is necessary for their description It is useful to di erentiate between two kinds of complex systems Complex systems of the rst kind with known simple dynamics but un known complex behavior In this case we aim at the analysis and classi cation of possible behavior Complex systems of the second kind or essentially complex systems with unknown dynamics and partially known complex behavior In this case our goal is to simplify description of these systems and if possible to nd the dynamics duch phys uni torun pl norbert phys uni torun pl So far only systems of the rst kind have been considered in the literature in particular the chaos theory deals with such systems Molecular complex structures of the second kind are too small and irregular for statistical mechanics and too large for fundamental theories to tackle Another example of complex system not covered by the quoted de nition is the structure of natural language Words and sentences have some regularity but it is very hard to nd the deep grammatical structure that will allow us to parse complex sentences Vocabularies are complex relatively large systems of information hard to analyze via mathematical means and apparently without simple underlying mechanism that could generate them Problem solving in arti cial intelligence leads to the representation spaces and decision trees that show combinatorial explosion thus leading to complex behavior or complex structure of their solution spaces Of course it may be that only trivially complex systems of the rst kind exist in nature and the essentially complex systems of the second kind are just arti cial con structions of the human mind Nevertheless at the present stage of scienti c inquiry it seems appropriate to develop also an approach that should allow for character ization of complex systems of the second kind where the dynamics is completely unknown or govern by too many degrees of freedom to handle it explicitly The goal of such theory would be to simplify the description of complex systems by nding a series of simpler descriptions approximations converging at the full complexity One source of inspiration for such theory comes from the theory of information devised by Shannon and others to measure the amount of information in an arbitrary data system Another approach is o ered by statistics in particular statistical theo ries of language The latest approach comes from the distributed storage of patterns in simpli ed neural networks Some interesting connections of complex systems of the second kind with in formation theory statistical approach and simplest models of neural networks are described below Information And Complexity Measures More than years after the de nition of information appeared in the landmark paper of Claude Shannon we still do not have a satisfactory de nition of infor mation that would be in accord with our intuition and that could unambiguously be applied to such concepts as biological information or linguistic information Di er ent de nitions or measures of information exist now including axiomatic de nition of Shannon information algorithmic information pragmatic information and cyclo matic information for details see The simplest approach to the quantitative de nition of information is based on combinatorics Interesting applications of combinatorial information to the estimation of entropy of a language have been reported Kolmogorov The entropy of words in a dictionary is considerably higher than the entropy of words in a literary text indicating that there are some constrains grammatical and stylistic in literary texts The second approach is based on probability It was introduced in the theory of information transmission by C Shannon and is based on his formula